Local Building Blocks for a Scalable Pub/Sub Infrastructure

A publish-subscribe (pub/sub) infrastructure [7] supports the dissemination of varied information to a large population of users with individual preferences. In this paper we focus on topic-based pub/sub systems, where users can subscribe to topics of interest, and receive all updates related to these topics. One notable application of such systems is real-time dissemination of trading data to stock brokers. Another example is a modern data center offering a large variety of application services that are accessed through the Internet. In such environments, the individual applications are typically replicated for performance and availability thereby creating overlapping multicast domains each of which is mapped into a separate pub/sub topic. Modern pub/sub applications have stringent scalability requirement. Information is often published by multiple sources, and disseminated to potentially tens or even hundreds of thousands of users, who may be geographically dispersed. Moreover, the information is categorized according to a rich collection of topics, possibly tens of thousands, and users typically subscribe to many topics. In order to deal with the huge number of users as well as with the system’s geographical scale, nodes are organized in a logical structure, e.g., a hierarchical [14] or other overlay organization [9, 6], where each node has a small number of neighbors. The main challenge in adapting existing overlay network technology for pub/sub applications is coping with the immense number of topics. It is possible to create a single overlay for all topics, where nodes subscribed to each topic form a connected sub-graph, as we did in SpiderCast [6, 5]. Although this significantly reduces the number of links per user, it does not solve all the difficulties stemming from the vast number of topics, for example, the difficulty of ensuring reliable message delivery in each group of